The IoT Sensor Streaming Warehouse project delivers a complete real-time data ingestion and analytics platform built fully on AWS. Using a streaming-first architecture, the solution ingests high-velocity sensor data (e.g., temperature, humidity, device location) via AWS Kinesis, processes it with AWS Lambda and AWS Glue, stores structured and raw data in Amazon S3, and serves analytics through Amazon Redshift. It enables sub-5-second end-to-end latency for critical alerts, handles millions of events per minute, and supports scalable, fault-tolerant, real-time and batch insights.
The pipeline follows a modern streaming design optimized for IoT workloads. Data is ingested into AWS Kinesis Data Streams, validated through AWS Lambda, ETL-processed with AWS Glue (PySpark), and stored in Amazon S3 as raw and processed data. Redshift acts as the analytical warehouse consuming enriched and aggregated datasets. CloudWatch provides monitoring, and X-Ray supports end-to-end tracing. This design ensures high throughput, reliability, real-time processing, and scalable long-term analytics.
Raw Zone (S3): Stores unmodified JSON sensor events from IoT devices.
Cleansed Zone (Glue / S3): Standardized and validated data, enriched with metadata, schema-verified, stored as Parquet.
Aggregated Zone (Redshift): Fact: sensor_readings (all processed sensor data); Dimensions: sensors, locations. Optimized using sort keys, distribution keys, and partitioning for fast queries.
Extract: IoT devices publish JSON events → collected by Kinesis → Lambda validates schema, nulls, and anomalies.
Transform: Glue PySpark ETL jobs perform cleansing, enrichment joins with metadata stored in S3, SCD-like updates, hourly/daily aggregations, and conversion to Parquet.
Load: Partitioned outputs written to S3 (year/month/day/hour). Redshift consumes data through Spectrum or via COPY loads.
Testing included unit tests for Lambda/Glue, integration tests for end-to-end flow, and load testing via JMeter (10k events/sec). Performance benchmarks ensured ETL under 2 seconds and uptime at 99.9%. Deployment followed CI/CD with CodePipeline, monitored cutover, and rollback procedures using Redshift snapshots.
Monitoring via dashboards and CloudWatch alarms for proactive issue detection. Targets include >99% pipeline reliability, Redshift utilization < 80%, and data alerts within 5 seconds. Cost control implemented via S3 Intelligent-Tiering and Redshift pause/resume.
.png)
.png)
.png)